--- title: RNN vs. Dense neural networks for time-series author: Bruce Meng date: '2018-06-17' slug: rnn-vs-dense-neural-networks-for-time-series categories: [] tags: - R - modelling ---
This is a continuation from my last post comparing an automatic neural network from the package forecast with a manual Keras model.
I used a fully connected deep neural network in that post to model sunspots. There’s another type of model, called a recurrent neural network, that has been widely considered to be excellent at time-series predictions. We’ll use the Gated Recurrent Units (GRU) model specifically. Let’s run through a comparison between a deep feed-forward neural network model established in the prior post with a GRU type of model.
We’ll reuse the sunspots dataset since it’s one of the better ones (it’s long and exhibits nice seasonal patterns).

And this is the testing data which we will test our models against:

First up is the dense network.
Step 1 - we will need to manually prepare the dataset into a format that Keras can understand. The code is a bunch of scaling, centering and turning the data from a tibble/data.frame to a matrix. I will skip showing that section as I suspect you’ll find it boring and it takes up quite a bit of room.
Step 2 - we can now construct a Keras model:
# Model params
units <- 256
inputs <- 1
# Create model
model.dense <- keras_model_sequential()
model.dense %>%
layer_dense(units = units,
input_shape = c(lookback),
batch_size = inputs,
activation = "relu") %>%
layer_dense(units = units/2,
activation = "relu") %>%
layer_dense(units = units/8,
activation = "relu") %>%
layer_dense(units = 1)
# Compile model
model.dense %>% compile(optimizer = "rmsprop",
loss = "mean_squared_error",
metrics = "accuracy")
Step 3 - we can now attempt to train the model:
## Model
## ___________________________________________________________________________
## Layer (type) Output Shape Param #
## ===========================================================================
## dense_1 (Dense) (1, 256) 37120
## ___________________________________________________________________________
## dense_2 (Dense) (1, 128) 32896
## ___________________________________________________________________________
## dense_3 (Dense) (1, 32) 4128
## ___________________________________________________________________________
## dense_4 (Dense) (1, 1) 33
## ===========================================================================
## Total params: 74,177
## Trainable params: 74,177
## Non-trainable params: 0
## ___________________________________________________________________________
Step 4 - we can now make predictions from the model:
## Predict based on last observed sunspot number
n <- 299 #number of predictions to make
predictions <- numeric() #vector to hold predictions
# Generate predictions, starting with last observed sunspot number and feeding
# new predictions back into itself
for(i in 1:n){
pred.y <- x[(nrow(x) - inputs + 1):nrow(x), 1:lookback]
dim(pred.y) <- c(inputs, lookback)
# forecast
fcst.y <- model.dense %>% predict(pred.y, batch_size = inputs)
fcst.y <- as_tibble(fcst.y)
names(fcst.y) <- "x"
# Add to previous dataset data.tibble.rec
data.tibble.rec.dense <- rbind(data.tibble.rec.dense, fcst.y)
## Recalc lag matrix
# Setup a lagged matrix (using helper function from nnfor)
data.tibble.rec.lag <- nnfor::lagmatrix(data.tibble.rec.dense$x, 0:lookback)
colnames(data.tibble.rec.lag) <- paste0("x-", 0:lookback)
data.tibble.rec.lag <- as_tibble(data.tibble.rec.lag) %>%
filter(!is.na(.[, ncol(.)])) %>%
as.matrix()
# x is input (lag), y is output, multiple inputs
x <- data.tibble.rec.lag[, 2:(lookback + 1)]
dim(x) <- c(nrow(x), ncol(x))
y <- data.tibble.rec.lag[, 1]
dim(y) <- length(y)
# Invert recipes
fcst.y <- fcst.y * (range.max.step - range.min.step) + range.min.step
# save prediction
predictions[i] <- fcst.y %>%
InvBoxCox(l)
predictions <- unlist(predictions)
}
Step 1 - we need to slightly tweak the data for GRU models since GRU expects a 3D tensor, instead of the 2D tensor used in the prior model.
Step 2 - we can now construct a Keras model:
# Model params
units <- 4
inputs <- 1
# Create model
model.GRU <- keras_model_sequential()
model.GRU %>%
layer_cudnn_gru(units = units,
input_shape = c(lookback, 1),
batch_size = inputs,
stateful = T,
return_sequences = T
) %>%
layer_dropout(0.2) %>%
layer_cudnn_gru(units = units/2,
stateful = T,
return_sequences = T) %>%
layer_cudnn_gru(units = 1,
stateful = T) %>%
layer_dropout(0.2) %>%
layer_dense(units = 1)
# Compile model
model.GRU %>% compile(optimizer = "rmsprop",
loss = "mean_squared_error",
metrics = "accuracy")
Step 3 - we can now attempt to train the model:
## Model
## ___________________________________________________________________________
## Layer (type) Output Shape Param #
## ===========================================================================
## cu_dnngru_1 (CuDNNGRU) (1, 144, 4) 84
## ___________________________________________________________________________
## dropout_1 (Dropout) (1, 144, 4) 0
## ___________________________________________________________________________
## cu_dnngru_2 (CuDNNGRU) (1, 144, 2) 48
## ___________________________________________________________________________
## cu_dnngru_3 (CuDNNGRU) (1, 1) 15
## ___________________________________________________________________________
## dropout_2 (Dropout) (1, 1) 0
## ___________________________________________________________________________
## dense_5 (Dense) (1, 1) 2
## ===========================================================================
## Total params: 149
## Trainable params: 149
## Non-trainable params: 0
## ___________________________________________________________________________
Step 4 - we can now make predictions from the model:
## Predict based on last observed sunspot number
predictions.GRU <- numeric() #vector to hold predictions.GRU
# Generate predictions.GRU, starting with last observed sunspot number and feeding
# new predictions.GRU back into itself
for(i in 1:n){
pred.y.GRU <- x.GRU[(nrow(x.GRU) - inputs + 1):nrow(x.GRU), 1:lookback, 1]
dim(pred.y.GRU) <- c(inputs, lookback, 1)
# forecast
fcst.y.GRU <- model.GRU %>% predict(pred.y.GRU, batch_size = inputs)
fcst.y.GRU <- as_tibble(fcst.y.GRU)
names(fcst.y.GRU) <- "x"
# Add to previous dataset data.tibble.rec
data.tibble.rec.GRU <- rbind(data.tibble.rec.GRU, fcst.y.GRU)
## Recalc lag matrix.GRU
# Setup a lagged matrix.GRU (using helper function from nnfor)
data.tibble.rec.lag <- nnfor::lagmatrix(data.tibble.rec.GRU$x, 0:lookback)
colnames(data.tibble.rec.lag) <- paste0("x-", 0:lookback)
data.tibble.rec.lag <- as_tibble(data.tibble.rec.lag) %>%
filter(!is.na(.[, ncol(.)])) %>%
as.matrix()
# x.GRU is input (lag), y.GRU is output, multiple inputs
x.GRU <- data.tibble.rec.lag[, 2:(lookback + 1)]
dim(x.GRU) <- c(nrow(x.GRU), ncol(x.GRU), 1)
y.GRU <- data.tibble.rec.lag[, 1]
dim(y.GRU) <- length(y.GRU)
# Invert recipes
fcst.y.GRU <- fcst.y.GRU * (range.max.step - range.min.step) + range.min.step
# save prediction
predictions.GRU[i] <- fcst.y.GRU %>%
InvBoxCox(l)
predictions.GRU <- unlist(predictions.GRU)
}
Ok let’s see some results! (Since we have 2 models, I’m also going to sneak in an ensemble model which is simply an average of the dense and GRU model predictions).